---
id: observability
title: Observability - Python SDK
sidebar_label: Observability
description: Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Learn to emit metrics, set up tracing, log from Workflows, and use custom Search Attributes.
slug: /develop/python/observability
toc_max_heading_level: 2
keywords:
  - client
  - code sample
  - developer-guide
  - guide-context
  - how-to
  - logging
  - python
  - python sdk
  - sdk
  - search attribute
  - workflow
tags:
  - Observability
  - Workflows
  - Search Attributes
  - Python SDK
  - Temporal SDKs
---

The observability section of the Temporal Developer's guide covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflows#workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Emit metrics](#metrics)
- [Set up tracing](#tracing)
- [Log from a Workflow](#logging)
- [Visibility APIs](#visibility)

## Emit metrics {#metrics}

**How to emit metrics**

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

Metrics can be scraped and stored in time series databases, such as:

- [Prometheus](https://prometheus.io/docs/introduction/overview/)
- [M3db](https://m3db.io/docs/)
- [statsd](https://github.com/statsd/statsd)

Temporal also provides a dashboard you can integrate with graphing services like [Grafana](https://grafana.com/docs/). For more information, see:

- Temporal's implementation of the [Grafana dashboard](https://github.com/temporalio/dashboards)
- [How to export metrics in Grafana](https://github.com/temporalio/helm-charts#exploring-metrics-via-grafana)

Metrics in Python are configured globally; therefore, you should set a Prometheus endpoint before any other Temporal code.

The following example exposes a Prometheus endpoint on port `9000`.

```python
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig

# Create a new runtime that has telemetry enabled. Create this first to avoid
# the default Runtime from being lazily created.
new_runtime = Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_address="0.0.0.0:9000")))
my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime)
```

## Set up tracing {#tracing}

**How to set up tracing**

Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows.

Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. If you need custom tracing specific for your use case, you should make use of context propagation to add tracing logic accordingly.

To configure tracing in Python, install the `opentelemetry` dependencies.

```bash
# This command installs the `opentelemetry` dependencies.
pip install temporalio[opentelemetry]
```

Then the [`temporalio.contrib.opentelemetry.TracingInterceptor`](https://python.temporal.io/temporalio.contrib.opentelemetry.TracingInterceptor.html) class can be set as an interceptor as an argument of [`Client.connect()`](https://python.temporal.io/temporalio.client.Client.html#connect).

When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker.
Spans are created and serialized through the server to give one trace for a Workflow Execution.

## Log from a Workflow {#logging}

**How to log from a Workflow**

Send logs and errors to a logging service, so that when things go wrong, you can see what happened.

The SDK core uses `WARN` for its default logging level.

You can log from a Workflow using Python's standard library, by importing the logging module `logging`.

Set your logging configuration to a level you want to expose logs to.
The following example sets the logging information level to `INFO`.

```python
logging.basicConfig(level=logging.INFO)
```

Then in your Workflow, set your [`logger`](https://python.temporal.io/temporalio.workflow.html#logger) and level on the Workflow. The following example logs the Workflow.

<div className="copycode-notice-container">
  <a href="https://github.com/temporalio/documentation/blob/main/sample-apps/python/your_loggers/your_workflow_dacx.py">
    View the source code
  </a>{' '}
  in the context of the rest of the application code.
</div>

```python
# ...
        workflow.logger.info("Workflow input parameter: %s" % name)
```

### Custom logger {#custom-logger}

Use a custom logger for logging.

Use the built-in [Logging facility for Python](https://docs.python.org/3/library/logging.html) to set a custom logger.

## Visibility APIs {#visibility}

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### Use Search Attributes {#search-attributes}

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/visibility#search-attribute).

- [Default Search Attributes](/visibility#default-search-attributes) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`).
  - A few [generic Custom Search Attributes](/visibility#custom-search-attributes) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service in the Temporal CLI or Web UI.
  - For example: `temporal operator search-attribute create --name CustomKeywordField --type Text`
    - Replace `CustomKeywordField` with the name of your Search Attribute.
    - Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList`
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `upsert_search_attributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/visibility#list-filter):
  - [In the Temporal CLI](/cli/operator#list-2)
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

Use the [list_workflows()](https://python.temporal.io/temporalio.client.Client.html#list_workflows) method on the Client handle and pass a [List Filter](/visibility#list-filter) as an argument to filter the listed Workflows.

<div className="copycode-notice-container">
  <a href="https://github.com/temporalio/documentation/blob/main/sample-apps/python/your_visibility/starter_dacx.py">
    View the source code
  </a>{' '}
  in the context of the rest of the application code.
</div>

```python
# ...
    async for workflow in client.list_workflows('WorkflowType="GreetingWorkflow"'):
        print(f"Workflow: {workflow.id}")
```

### How to set custom Search Attributes {#custom-search-attributes}

After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create`or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

Use `SearchAttributeKey` to create your Search Attributes. Then, when starting a Workflow execution using `client.start_workflow()`, include the Custom Search Attributes by passing instances of `SearchAttributePair()` containing each of your keys and starting values to a parameter called `search_attributes`.
If you had Custom Search Attributes `CustomerId` of type `Keyword` and `MiscData` of type `Text`, you could provide these starting values:

```python
customer_id_key = SearchAttributeKey.for_keyword("CustomerId")
misc_data_key = SearchAttributeKey.for_text("MiscData")

handle = await client.start_workflow(
    GreetingWorkflow.run,
    id="search-attributes-workflow-id",
    task_queue="search-attributes-task-queue",
    search_attributes=TypedSearchAttributes([
        SearchAttributePair(customer_id_key, "customer_1"),
        SearchAttributePair(misc_data_key, "customer_1_data")
    ]),
)
```

In this example, `CustomerId` and `MiscData` are set as Search Attributes.
These attributes are useful for querying Workflows based on the customer ID or the date the order was placed.

### Upsert Search Attributes {#upsert-search-attributes}

You can upsert Search Attributes to add or update Search Attributes from within Workflow code.

To upsert custom Search Attributes, use the [`upsert_search_attributes()`](https://python.temporal.io/temporalio.workflow.html#upsert_search_attributes) method to pass instances of `SearchAttributePair()` containing each of your keys and starting values to a parameter to a `TypedSearchAttributes()` object:

```python
workflow.upsert_search_attributes(TypedSearchAttributes([
    SearchAttributePair(customer_id_key, "customer_2")
]))
```

### Remove a Search Attribute from a Workflow {#remove-search-attribute}

To remove a Search Attribute that was previously set, set it to an empty array: `[]`.

```python
workflow.upsert_search_attributes(TypedSearchAttributes([
    SearchAttributePair(customer_id_key, [])
]))
```
